1. Field of the Invention
The present invention generally relates to an image signal format converter and an associated method of performing the conversion. More particularly, the invention relates to a format converter, and method, that performs frame rate conversion and de-interlacing using a bi-directional motion vector. The present application is based on Korean Patent Application No. 00-32388 filed on Jun. 13, 2000, which is incorporated herein by reference.
2. Description of the Related Art
Generally, in a PC or a high definition TV (HDTV), conversion of a format such as a frame rate and/or de-interlacing is needed to exchange signals utilizing various signal standards.
FIG. 1 is a block diagram of a conventional frame rate conversion apparatus.
Referring to FIG. 1, an image dividing unit 110 divides an image into a changed region and an unchanged region for efficient motion estimation as shown in FIG. 2. The unchanged region is divided again into a covered region, an uncovered region, a background, and an object.
A motion estimating unit 120 generates the motion vector of a block using a block matching algorithm which is generally known in the art of video coding. In a representative conventional block matching algorithm, a motion vector is determined for each block based on the assumption that pixels in a predetermined-size block, as shown in FIG. 3, moved without rotation, magnification, or reduction. In FIG. 3, it is assumed that the motion vector of an N×N size base block located on arbitrary coordinates (xc, yc) in the current frame (fc) is estimated in a range of ±P pixels from the previous frame (fp). Then, the search range in the previous frame is (N+2P)×(N+2P). Therefore, the motion vector is determined at a location which has the maximum correlation among the total (2P+1)2 candidate locations.
At this time, the difference between the base block in the current frame and the candidate block in the previous frame is calculated as a mean absolute difference (MAD) by the following equation 1:                     MAD        =                              1                          N              2                                ⁢                                    ∑                              i                =                1                            N                        ⁢                                          ∑                                  j                  =                  1                                N                            ⁢                                                                                                          f                      c                                        ⁡                                          (                                                                                                    x                            c                                                    +                          i                                                ,                                                                              y                            c                                                    +                          j                                                                    )                                                        -                                                            f                      p                                        ⁡                                          (                                                                                                    x                            c                                                    +                          i                          +                          m                                                ,                                                                              y                            c                                                    +                          j                          +                          n                                                                    )                                                                                                                                              (        1        )            
Here, the motion vector of the final block is determined at a location (m, n) of a search range for which the mean absolute difference between the current block and the candidate block is a minimum.
A spatiotemporal smoothing unit 130 refines an inappropriate motion vector obtained in the motion estimating unit 120, and refines the smoothness of a motion vector as shown in FIG. 4.
A motion-compensated interpolation unit 140 finds a forward motion vector for the previous frame and the next frame of an image to be interpolated and, using the obtained motion vector, performs bi-directional interpolation according to region classification information generated in the image dividing unit 110. At this time, as shown in FIG. 5, the motion-compensated interpolation using a forward motion vector generates overlaps, where blocks are overlapped because two or more motion vectors are assigned in a frame to be interpolated, and holes where a motion vector is not assigned. These overlaps and holes directly affect the picture quality of an image by causing a degradation in the picture quality. Also, since these overlaps and holes have irregular shapes, they should be handled on a pixel-by-pixel basis. Therefore, for the conventional apparatus, complicated signal processing processes and complicated hardware are necessary to remove the overlaps and holes.
Also, for an ordinary television video signal, a frequency band is compressed using an interlacing method in which two fields form a frame. Recently, in a PC or an HDTV, for which the displays generally use a progressive scanning method to display an interlaced image, empty image lines, due to de-interlacing, are generated by an arbitrary method for progressive scanning.
FIG. 6 is a conceptual basic diagram of ordinary de-interlacing.
Referring to FIG. 6, de-interlacing changes a field containing only an odd-numbered pixel, or only an even-numbered pixel, into a frame. At this time, an output frame F0({right arrow over (x)}, n) is defined by the following equation 2:                                           F            0                    ⁡                      (                                          x                →                            ,              n                        )                          =                  {                                                                                          F                    ⁡                                          (                                                                        x                          ⇀                                                ,                        n                                            )                                                        ,                                                                              (                                                            y                      ⁢                                                                                           ⁢                      mod                      ⁢                                                                                           ⁢                      2                                        =                                          n                      ⁢                                                                                           ⁢                      mod                      ⁢                                                                                           ⁢                      2                                                        )                                                                                                                                                F                      i                                        ⁡                                          (                                                                        x                          →                                                ,                        n                                            )                                                        ,                                                            otherwise                                                                        (        2        )            
Here, {right arrow over (x)} means a spatial location, and n is a field number. F({right arrow over (x)}, n) is an input field, and Fi({right arrow over (x)}, n) is a pixel to be interpolated.
FIG. 7 is a 3×3 window for applying an edge-based line averaging (ELA) de-interlacing algorithm which does not use motion compensation.
Referring to FIG. 7, ELA de-interlacing uses the correlation between pixels considering direction (x, y) from the location of a pixel to be interpolated as shown in equation 3, below. That is, the mean value of pixels neighboring a pixel to be interpolated and pixels to be interpolated in the previous field and the next field of the field to be interpolated is output, as follows:                                           F            0                    ⁡                      (                                          x                →                            ,              n                        )                          =                  {                                                                                                                F                      ⁡                                              (                                                                                                            x                              →                                                        -                                                                                          u                                →                                                            x                                                        -                                                                                          u                                →                                                            y                                                                                ,                          n                                                )                                                              +                                                                  F                        ⁡                                                  (                                                                                                                    x                                →                                                            +                                                                                                u                                  →                                                                x                                                            +                                                                                                u                                  →                                                                y                                                                                      ,                            n                                                    )                                                                    /                      2                                                        ,                                                                                                  if                    ⁢                                                                                   ⁢                                          min                      ⁡                                              (                                                  a                          ,                          b                          ,                          c                                                )                                                                              =                  a                                                                                                                                                F                      ⁡                                              (                                                                                                            x                              →                                                        -                                                                                          u                                →                                                            x                                                        +                                                                                          u                                →                                                            y                                                                                ,                          n                                                )                                                              +                                                                  F                        ⁡                                                  (                                                                                                                    x                                →                                                            +                                                                                                u                                  →                                                                x                                                            -                                                                                                u                                  →                                                                y                                                                                      ,                            n                                                    )                                                                    /                      2                                                        ,                                                                                                  if                    ⁢                                                                                   ⁢                                          min                      ⁡                                              (                                                  a                          ,                          b                          ,                          c                                                )                                                                              =                  b                                                                                                                                                F                      ⁡                                              (                                                                                                            x                              →                                                        -                                                                                          u                                →                                                            y                                                                                ,                          n                                                )                                                              +                                                                  F                        ⁡                                                  (                                                                                                                    x                                →                                                            +                                                                                                u                                  →                                                                y                                                                                      ,                            n                                                    )                                                                    /                      2                                                        ,                                                            otherwise                                                                        (        3        )                                                                                    here                ,                                  a                  =                                    ⁢                                                                                                        F                        ⁡                                                  (                                                                                                                    x                                →                                                            -                                                                                                u                                  →                                                                x                                                            -                                                                                                u                                  →                                                                y                                                                                      ,                            n                                                    )                                                                    -                                              F                        ⁡                                                  (                                                                                                                    x                                →                                                            +                                                                                                u                                  →                                                                x                                                            +                                                                                                u                                  →                                                                y                                                                                      ,                            n                                                    )                                                                                                                                                                                                                b                =                                ⁢                                                                                              F                      ⁡                                              (                                                                                                            x                              →                                                        -                                                                                          u                                →                                                            x                                                        +                                                                                          u                                →                                                            y                                                                                ,                          n                                                )                                                              -                                          F                      ⁡                                              (                                                                                                            x                              →                                                        +                                                                                          u                                →                                                            x                                                        -                                                                                          u                                →                                                            y                                                                                ,                          n                                                )                                                                                                                                                                                      c                =                                ⁢                                                                                              F                      ⁡                                              (                                                                                                            x                              →                                                        -                                                                                          u                                →                                                            y                                                                                ,                          n                                                )                                                              -                                          F                      ⁡                                              (                                                                                                            x                              →                                                        +                                                                                          u                                →                                                            y                                                                                ,                          n                                                )                                                                                                                                                  ⁢                                                                    
FIG. 8 is a conceptual diagram for explaining an ordinary time-recursive (TR) de-interlacing method.
Referring to FIG. 8, TR de-interlacing using a motion vector assumes that the previous field (n−1) is perfectly de-interlaced, and the missing data of the current field (n) is compensated with a motion vector. A pixel to be interpolated can be the original pixel of the previous field, or a previously interpolated pixel. Therefore, a pixel to be interpolated can be expressed as the following equation 4:                                           F            0                    ⁡                      (                                          x                →                            ,              n                        )                          =                  {                                                                                          F                    ⁡                                          (                                                                        x                          ⇀                                                ,                        n                                            )                                                        ,                                                                                                  (                                                                  y                        ⁢                                                                                                   ⁢                        mod                        ⁢                                                                                                   ⁢                        2                                            =                                              n                        ⁢                                                                                                   ⁢                        mod                        ⁢                                                                                                   ⁢                        2                                                              )                                    ,                                                                                                                          F                    ⁡                                          (                                                                                                    x                            →                                                    -                                                      d                            ⁡                                                          (                                                                                                x                                  →                                                                ,                                n                                                            )                                                                                                      ,                                                  n                          -                          1                                                                    )                                                        ,                                                            otherwise                                                                        (        4        )            
However, since the ELA de-interlacing method does not use motion compensation, flickering occurs in regions where movement exists and, since the TR de-interlacing method is continuously de-interlaced, an error that occurs in an arbitrary field can be propagated to other fields.